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ELECTRONIC BILLBOARD: A METHOD OF ADVERTISING USING EXISTING 
TELEVISION TRANSMISSION FACILITIES. 



This invention relates to a method of advertising using existing television 
transmission facilities in which the advertiser selects certain areas in a video scene, 
recognizes these areas using computer based technology, and replaces the selected areas by 
inserting an image or images of the advertisers choosing. The method of this invention is 
especially applicable when the recognizing of an area and its subsequent replacement is 
done in real time, and where the insert is blended into the original video in such a way that 
it appears real to the television viewing audience. The inserted advertising or messages 
may have their impact on the television audience enhanced by making them respond, via 
motion, colour change or other animation, to sound and/or images from the original scene. 
Although not limited thereto, the present invention is particularly suitable for the display of 
advertising in live, televised sporting events. It may readily be adapted to all formats of 
video presentation, including future developments such as high definition television 
formats. 

The essence of the invention is the method of altering video images by selection all 
or part of an object or objects within a 1st video image, then recognizing that same part or 
all of an object in each or any of a subsequent stream of video images, and using the 
position of that object or part of it, as a reference to accurately insert and position a 2nd 
still or video image into each or any of the stream of video images. The method allows 
advertisers to add or merge images, messages or slogans to preexisting video images in 
such a way that they appear to be part of the original image, even when the original video 
stream is of a live event, being distributed in real time. 

Existing technology for merging video images includes the "blue screen" or 
"chromo-key" techniques, currently used in broadcast industry to show weather forecasters 
in front of weather maps. In this technology, the presenter is filmed in front of a screen of 
a particular, pre-defined colour, which is usually blue (hence the name). When merging 
the two images, the technique amounts to substituting the second image, which in this 
example would be the weather map, at all points in the first image ( the weather person 
image) where the signal is of the predetermined blue. The method has two short comings. 
Firsdy, the presenter must not have any item on them that is of the blue that will be 
substituted, else the second image will be merged in at that point, resulting in a weather 
forecaster who appears to have holes in them. Secondly, and more important for the 



advertising application, there is no correspondence between the two images. They are 
merely superimposed, which is why weather forecasters make sweeping and general 
motions when indicating places. 

The advertising method of this invention requires very precise positioning of a new 
image into an existing image. This requires pattern recognition of preselected features, such 
as the goal posts in a scene of a soccer match. These features can then be used to locate 
the position, size and perspective of an artificial electronic billboard, which is added to the 
video image and appears to the end user as if it were part of the original scene. This is 
described in our application number 9003275.6 to the Patent Office, London, entitled 
"Electronic Billboard", filed by R J. Rosser and M. Leach, from which this application 
claims priority, and subsequent enhancement to that in UK Patent application number 
9019770.8. 

To effectively apply the method of selection, recognition and substitution to 
broadcast television of live events, an electronic device is required that detects part or all of 
an object or objects within a video image, and uses the position of that object or part of it, 
as a reference to accurately position another still or video image into the original in such a 
way that the final composite image appears as if it were of the original scene. That is, the 
added image is seamlessly and realistically incorporated into the original even when the 
original scene is moved, panned, magnified, zoomed or otherwise alters in size or 
perspective. All this must be done in real time, or with the processing delay being 
sufficiently small that it can be applied to events being transmitted live. The most 
appropriate - possibly the only way - to do this amount of computation on a video image in 
such a short time with existing or foreseeable technology, is to use the "Burt Pyramid", a 
well known method of processing, analyzing and/or synthesizing signals described in U.S. 
Patent application No. 4,674,125 by Dr. Peter J. Burt, entitled "Real-time hierarchical 
pyramid signal processing apparatus" and discussed in publications such as "Fast 
Algorithms for Estimating Local Image Properties" by Peter J. Burt, Computer Vision, 
Graphics and Image Processing 21, 368-382 (1983) and "Pyramid-based Extraction of 
Local Image Features with Applications to Motion and Texture Analysis" by Peter J. Burt, 
SPIE, Vol. 360, 114-124. 

The "chromo-key" or "blue screen" technology may be useful as an adjunct to the 
pattern recognition method, being one way of distinguishing non-replaceable, foreground 
pixels from replaceable, background pixels in certain applications. For instance, when a 
message is merged on to the field of play of some sport, such as tennis, in most 
applications it is necessary to make sure that only the tennis court and not the players or 
their equipment are replaced. This may be done in several ways, one of which is to use 
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the fact that the court is green, and only replace pixels in the merged image that arc within 
the colour range of the court. Alternate methods of distinguishing between replaceable and 
non - replaceable pixels include storing an image of the unoccupied court, and 
subsequently using that as a comparison to determine what is player and equipment and 
what is background. Differential motion between objects or markings known to be on the 
court and equipment and players moving over it can also be used as a basis for 
distinguishing between replaceable and non-replaceable pixels. 

The potential uses of such a method of advertising include, but are not limited to, 
advertising in sporting events such as soccer or rugby where there is continuous action, 
and no convenient break for the conventional television advertisement. By making the 
advertising a discrete part of the action, the game would not have to be interrupted. Even in 
sports where there are breaks for advertising, the method described here is of benefit to 
advertisers because the message can be displayed while the audience's attention is focussed 
on the action. It can also be used to enhance such games as American football by artificially 
including markings for the line of scrimmage and the first down line. Additionally, the 
message can be changed during the game, it can be animated and it can be different for 
different target audiences in different locations. Specific examples of applications of the 
method include, but are not limited to: 

• placing advertising in or around the court of play, 

• placing advertising on a bowling ball, or a basket ball, 

• placing advertising on the bowling ally lane, or parts of a basketball or tennis 
court, including the backboard of a basket ball court or the net of a tennis court, 

• having recognizable patches on individual players in any sport which could then 
be replaced with advertising, which could be different in different parts of the 
county or world, 

• and special video effects such as "exploding scoreboards" at stadiums where 
such facilities don't exist 

Although the pattern recognition software using the "Burt pyramid" algorithms is 
fast, it still takes a finite time to operate. It may, therefore, be important to introduce a time 
delay into the system. This may be of the order of one or two frames, though it may be 
longer. This can be done by including a frame store, or other means of temporarily 
locating a small number of the video screens or frames. The possibility of incorporating a 
time delay unit into the electronic billboard system is an important part of this invention. 



The method of this invention can be further enhanced by allowing the inserted 
video image to react to some aspect of the original scene, such as the sound. For instance, 
the inserted slogan could change colour, vibrate or move as the sound level from the 
stadium reached a certain level. Or it could have a colour which was related to the sound 
level - a colour "thermometer" of the sound level In a more sophisticated version, the 
image change may be keyed to recognition of a particular word by the commentators, using 
voice recognition software and hardware. For instance the slogan could change when a 
particular player - or team or when the product name - is mentioned 

Such a system will greatly enhance the effectiveness of electronic billboard method 
of advertising by linking visual and audio ques. With the development of the appropriate 
technology the concept can be extended to include visual enhancement of the visual pattern. 
For instance a sophisticated pattern recognition system may recognize the presence of a 
particular player and use that to animate or change the inserted image in some suitable 
fashion. It may also be effective to allow the commentator or other supervisor to control 
the change in the scene manually, so that it would occur when, for instance, a hoop was 
scored in a basketball match. 

A specific embodiment of the invention will now be described with reference to the 
attached drawings in which :- 

Figure 1 is a schematic view of the technology necessary to implement a version of the 
electronic billboard method of advertising. 

A sporting event is in progress on a field of play 12, which may for example be a 
tennis match. This sporting event is recorded by means of television cameras 10 and sound 
equipment 20 for immediate or subsequent distribution to viewers on television sets 18. 
This distribution of the television signal is done via a suitable transmission and reception 
system 26, which may be wireless broadcast or a cable network, both of which are well 
known methods of distributing television signals. Before distribution of the signal to the 
television sets 18, or at one or more suitable junctures in the transmission chain, the image 
of the event 12 as recorded by the video cameras 10 is altered to incorporate an advertising 
message, which may be a slogan or image. Such alteration may take place at the 
originating site of the event, at a network broadcast center, at a local broadcast center, at a 
local broadcasting station, at a cable head-end or any other site where video signals are 
disseminated. This added message is produced by a separate video source 16 which may 
be a prerecorded video tape, and is merged into the video stream by the image synthesizer 



22 using data from the image analyzer 14 and the frame storage and delay device 20. The 
essence of this method of advertising is the incorporation of the added message from the 
secondary source 16 into the main image recorded by the camera 10 in such a way that to 
the end-viewer looking at the television picture produced on the television 18 the added 
logo appears as if it were actually on the court 12, even when the camera 10 is panned 
and/or zoomed to follow the action of the game occurring on the court 12. This seamless 
merging of the advertising message or image into the video images from the camera 10 is 
accomplished using the computer-like hardware and software represented by the image 
analyzer 14, the image storage and delay box 20 and the synthesis and merging box 22. 
The image analyzer 14, synthesizer 22, delay unit 20, 2nd video source 16 and the audio 
analyzer 24 taken altogether, constitute the enabling technology or device that needs to be 
added to existing television transmission systems in order to implement the Electronic 
Billboard method. Among other features, the image analyzer 14 includes the operator 
interface equipment (i.e. keyboard, lightpen, and/or hand operated mouse) to allow the 
operator to select landmarks in the original scene. The image analyzer 14 and synthesizer 
22 are electronic or optical processors and/or the software to drive such devices, capable of 
applying the well known "Burt Pyramid" techniques to the incoming video images in real 
time. (As described in US Patent 4,674,125, the Burt Pyramid algorithm uses a particular 
sampling technique for analyzing a relatively high resolution original image in to a 
hierarchy of N (where N is a plural integer) separate component images, in which each 
component image is a Laplacian image comprised of a different octave of the spatial 
frequencies of the original image, plus a remnant Gaussian image. The term pyramid 
relates to the successive reduction in spatial frequency bandwidth and sample density of 
each of the hierarchy of component images in going from the highest octave component to 
the lowest octave component image. The Burt Pyramid is used as it appears to be the most 
computationally effective method of dealing with visual images that are intended for 
viewing by humans. Other techniques of video processing may become available that 
allow the implementation of the Electronic Billboard method of advertising). The technique 
is used to accomplish three things. Firsdy the operator uses the video image of the court 12 
taken by the camera 10 to select areas of the court, which may be specific markings such as 
the tramlines of the tennis court, or the net. These markings are then recognized by the 
analyzer 14 in subsequent video images of the court 12 and the action occurring on it, and 
used by the synthesizer 22 to locate, position and orient, including the correct 
magnification, the logo from the second video source 16 so that it appears to be part of the 
original scene to the viewers of the television set 18. The logo added from 16 can be made 
to appear as part of the background - and not interfere with objects or people in the 
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foreground - by only allowing it to key over specific colours or range of colours, such as 
the colour of the court in a tennis match. (Or by not keying over specific colours, such as 
the clothing and flesh colours of the players). The distinction between foreground (which 
in many applications must not be obscured by the added image) and background (which 
must be replaced by the added image) may also or alternatively be identified by noise, 
contrast, or other characteristic of the original image. The resultant effect would be to have 
an unimpeded view of the match, with advertising discretely, yet conspicuously added 
The principle could be extended to use numbers, and other objects, including players 
themselves, as cues as to where to locate the electronic billboard. 

In order to give the analyzer 14 time to process the signal, the signal is feed into the time 
delay 20, which may be a frame store, or a multi-frame store. The frame store may be any 
memory medium that has sufficiently fast read/write times. The amount of delay and 
storage required by 20 will depend on the speed of the analyzer 14 and the quality of the 
image from the video camera 10. It will be larger for high definition format video than for 
current broadcast quality video, for instance. 

In addition to the video image of the scene 12 recorded by the cameras 10, the sound 
from the event is recorded by microphones 28 and added to the broadcast stream in the usual 
fashion. However by linking a suitable audio processor 24 into the audio stream from the 
microphone 20, the sound can be used to control some aspect of the image inserted by the 
second video source 16. In the simplest embodiment the audio processor 24 would only 
respond to the total volume form the microphone 20, and alter the added logo according to 
that For instance the colour of the logo could be keyed to the sound level, or the logo could 
be made to vibrate when the sound intensity reached a certain threshold. This could all be 
done by software control of microprocessors or by suitable electronic hardware. In a more 
sophisticated version, the audio processor 24 could be a voice recognition system, as are 
becoming routinely available. This would recognize, for instance, particular words spoken 
by say the commentator, and make the inserted video respond in a suitable fashion. For 
instance, the logo could change colour, do a flip, vibrate move or other wise change 
whenever the name of a particular player, team or even the product being advertised was 
mentioned by the commentator. It could even respond to particular words or phrases likely 
to be uttered by the commentator. 



CLAIMS 



1 A method of altering video images by selecting part or all of an object or objects 
within a 1st video image, then recognizing that same part or all of an object in each or any 
of a subsequent stream of video images, and using the position of that object or part of it, 
as a reference to accurately insert and position a 2nd still or video image into each or any of 
the stream of video images. 

2. The method of 1, used for the purposes of advertising on a television distribution 
system. 

3. A method as in 1 or 2, as performed by an electronic or optical processor and/ or 
software. 

4. A method as in 1, 2 or 3 in which the electronic or optical device and/or software 
enables the final, composite image to be such that the inserted image merges with the 
original and changes shape and hue in the same manner and to the same degree as the 
original image when it is moved, panned, magnified, zoomed or otherwise alters in size or 
perspective. 

5. A method as in 1,2,3 or 4 in which the enabling device allows the method to be 
implemented in real time, and/or with a delay of less than 1 second. 

6 A method as in 1,2,3 or 4 in which the enabling device used incorporates a time 
delay circuit , or stores, allowing up to 10 seconds time for the electronic device to 
compute the position of the objects and insert the second still or video image into place. 

7 A method as in 1,2,3 or 4 in which the enabling device used incorporates a time 
delay circuit , or stores, allowing up to one minutes time for the electronic device to 
compute the position of the objects and insert the second still or video image into place. 

8. A method as in 1,2, 3, 4, 5, 6 or 7 in which the enabling device allows the 
insertion of a multiplicity of still or video images or mixture thereof. 

9. A method as in 1, 2, 3, 4, 5, 6, 7 or 8 in which the enabling device uses colour 
levels to distinguish non-replaceable pixels, from replaceable, pixels. 
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10. A method as in 1, 2, 3, 4, 5, 6, 7 or 8 in which the enabling device uses motion to 
distinguish non-replaceable pixels, from replaceable, pixels. 

11. A method as in 1, 2, 3, 4, 5, 6, 7 or 8 in which the enabling device uses a 
previously stored video image of the original scene as a reference to distinguish non- 
replaceable pixels, from replaceable, pixels. 

12. A method as in 1, 2, 3, 4, 5, 6, 7, 8, 9, 10 or 11 in which the enabling device 
monitors the sound from the original scene or video of it, and uses that to change the 
inserted or merged still or video image. 

13. A method as in 12 which the enabling device uses voice recognition techniques to 
make the characterization specific to a particular word or set of words or specific sound. 

14. A method as in 12 in which the enabling device allows a controller to activate the 
change of the inserted or merged still or video image. 

15. A television or video recording unit or system using a method as described in any 
or all of claims 1 to 14 

16. A television or video broadcast or transmission unit using a method as described in 
any or all of claims 1 to 14 

17. A television or video distribution system using a method as described in any or all 
of claims 1 to 14. 

18. A method substantially as described herein with reference to 
Figure 1 of the accompanying drawing. 
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ABSTRACT 

ELECTRONIC BILLBOARD: A METHOD OF ADVERTISING USING EXISTING 
TELEVISION TRANSMISSION FACILITIES. 



This invention is a method of altering video images to enable the addition of images, 
messages or slogans in such a way that they appear to be a pan of the original image. A 
major use of the method is anticipated to be for advertising, using existing television 
transmission facilities. The method uses an electronic or optical processor and/or software 
to select part or all of an object or objects within a video image, and then recognizes that 
object in a subsequent stream of video images, and uses the position of that object or part of 
it, as a reference to accurately insert another still or video image into the stream of video 
images. This is accomplished so that the inserted image is seamlessly and realistically 
incorporated into the original, even when the original scene is moved, panned, magnified, 
zoomed or otherwise alters in size or perspective. The resultant composite image appears to 
the television viewer to all be real. 
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